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ABSTRACT 

This proposal describes a framework and strategy for 
evaluating the impact of World Bank Institute (WBI) programs. Each section 
details a relevant aspect of the evaluation strategy and framework, their 
goals and strategies, and how they relate to the broader goals and mission of 
the WBI and the World Bank. The evaluation proposal focuses on WBI client ' 
learning programs that have been implemented for at least 2 years and whose 
effects include impacts on individuals, countries, policies, laws, and 
regulations. The evaluation framework outlines a strategy for measuring 
outcomes in five areas, and the conceptual model recognizes the multilevel 
structure and multiple results of WBI programs through the inclusion of 
inputs, program description and implementation, outputs, outcomes, impacts, 
and exogenous factors, both direct and indirect. Five of 15 WBI programs were 
selected for in-depth impact evaluation in fiscal year 2002, and a random 
sample of abut 10% of their activities will be chosen for impact evaluations. 
For each of the five programs, this proposal plans the evaluation of 
activities that are of longer duration, rather than one-time or short 
offerings. A variety of data sources will be used to answer key research 
questions, and mixed methods will be used for data analysis. Findings will be 
disseminated as appropriate to a number of audiences. (SLD) 
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Impact Evaluation Framework and Strategy 



1. Background and Rationale 

1 . 1 Purpose of Impact Evaluation Framework 

This proposal describes a framework and strategy for evaluating the impact of the World Bank 
Institute’s (WBI) programs. Each section details a relevant aspect of the evaluation strategy and 
framework, their goals and strategies, and how they relate to the broader goals and mission of 
WBI and the World Bank (WB). 

1.2 Mission of the World Bank Institute 

The stated mission of WBI is to “Create learning opportunities for countries, Bank staff and 
clients, and people committed to poverty reduction and sustainable development.” 1 Key to this 
mission is WBI’s specific goal of promoting knowledge and learning for a better world. As the 
learning arm of the World Bank, WBI “supports the World Bank’s learning and knowledge 
agenda by providing learning programs and policy advice in the areas of environment and natural 
resources, economic policy for poverty reduction, governance, regulation and finance and human 
development.” 2 

Active in 150 countries, WBI delivers nearly 600 learning programs for its clients and reaches 
48,000 participants annually through face-to-face, open and distance learning in the five areas 
listed below. 3 Figure 1 displays the total number of training days for WBI programs for FY99-01 
segmented into the major thematic areas. 4 



Figure 1 . Thematic Distribution of WBI Programs, FY 1999-2001. 




□ FY 1999 

□ FY 2000 

□ FY 2001 



Environment 
and Natural 
Resources 



Economic 
Policy and 
Poverty 
Reduction 



Governance, Human 
Regulation and Development 
Finance 



Knowledge 
Networks and 
Distance 
Learning 



Other 



1 From WBFs brochure; “About the World Bank Institute.” 

2 Ibid 

3 Ibid 

4 Source: WBI Annual Reports, 1999-2001. Originally, FY99 data was used in the selection process of which WBI programs 
would be evaluated for impacts. 
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1.3 WBI Program Areas 



The focus of WBI’s work has shifted from delivering activities through sectoral divisions 
mapped to networks to delivering them through fifteen coordinated programmatic areas. This 
institutional change was implemented in FY01 in order to better link the WBI activities to the 
Bank’s overall development strategy, to improve the coordination of WBI team efforts that 
deliver training offerings in the same subject areas, to solidify WBI’s work with the entire Bank 
including the regions and networks, and to maximize the cost-effectiveness of WBI efforts 
through team oriented approaches to program delivery. The thematic program areas are: 



1 . 

2 . 

3. 

4. 

5. 

6. 

7. 

8 . 

9. 

10 . 
11 . 
12 . 

13. 

14. 

15. 



Sustainable Development: Environmental Management 

Sustainable Development: Natural Resources Management 

Knowledge for Development 

Education 

Good Governance 

Market Solutions for Development 

Attacking Poverty 

Community Empowerment and Social Inclusion (new) 
Public Finance, Decentralization and Urban 
Finance and Banking 
Globalization and Macroeconomics 
HIV/AIDS (new) 

Health and Population 

Social Protection 

Core Operational Programs (new) 



The number of offerings delivered by the Thematic Programs in FY 01 (programs offered by 
divisions were mapped into the new thematic areas) and that are planned to be delivered in FY 
02 is shown in Figure 2. 5 Three of the programs are new or relatively new and offered few, if 
any, activities in FY01. 




5 Source: WBI annual reports and WBIRO status reports. 
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Figure 2. WBI Activities by Program Areas, 
FY 01-02 




□ FY 01 

□ FY 02 (plan) 



2. Objectives of Impact Evaluations 

This evaluation proposal focuses on WBI client learning programs that have been implemented 
for at least two years and whose effects include impacts on individuals, countries, policies, laws 
and regulations. The guiding principles for assessing the impacts of the program are closely 
linked to the logical framework (logframe) design used in conceptualizing and implementing the 
development programs. The logframe approach helps to identify indicators for evaluating the 
impacts of the programs. It also helps in examining whether or not the program has achieved its 
stated objectives. 

The objective of impact evaluations of WBI programs is to determine how the programs 
contribute to institutional change at the country level, through changes in participants’ attitudes, 
learning and behaviors. Additionally, the achievement of the stated goals of the programs and 
their contribution to the improvement of development indicators in the targeted sectors of client 
countries are also seen as major sources of impacts. 



3. Impact Areas and Focus 

3.1 Defining Impact of WBI Programs 

Training provided by WBI under its fifteen programs is aimed at capacity building in the public 
sector. This enhanced capacity is expected to produce improvements in public policy design and 
implementation, leading to institutional change and improvements in the country’s development 
indicators. 

3.2 Focus of Impact Evaluation 

Our proposed evaluation of the impacts of WBI programs follows a logical sequence of 



measuring acquired knowledge, the application of this new knowledge, and its effects on 
institutions and public policy decisions. In this manner, impact evaluation recognizes the 
continuum of OUTPUTS - OUTCOMES -IMPACTS. The effectiveness of learning that is 
assessed through Levels 1 and 2 evaluations is supplemented with an assessment of behavioral 
change and performance enhancement through Level 3 evaluation and by the assessment of 
institutional change and improved policy design and implementation through Level 4 
evaluations. (See Figure 3.). This continuum is the foundation for the framework design shown 
in Annex 1. 



Figure 3. Continuum of Results of WBI Programs 




Levels 1 & 2 Level 3 Level 4 



4. Evaluation Framework 

4.1 Approach to Impact Evaluation 

WBI training programs for client countries are interventions aimed at changing existing 
conditions in public policy. These interventions cover a wide range of immediate objectives: 
from simple skills development and knowledge sharing to more complex goals of awareness 
raising and consensus building. Learning gains (level 2) by participants translate to certain 
attitudinal and behavioral changes (level 3) which can be attributed directly to the training 
intervention and are measurable. The difficulty with assessing impacts of WBI programs on 
public policy (level 4)lies with the complexity of stages that separate the level 3 outcomes from 
the level 4 impacts. Policy outcomes are the products of a multitude of factors and events; it is 
very difficult to isolate the true impact of WBI training as the cause of a change in public policy. 

4.2 Areas of Impact Evaluation 

The evaluation framework, as displayed in Annex 1, outlines a strategy for measuring outcomes 
of WBI training activities in the following areas: 

1. Relevance and effectiveness of the program’s design and its adequacy in meeting locally 
defined needs; 

2. Effectiveness of the training program’s content and delivery through the assessment of 
learning levels of participants; 

3. Effects on institutional capacity building in the countries; 

4. Effects on the design and implementation of public policy; 

5. Changes in sectoral and national development indicators; 
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Additionally, the framework in Annex 1 recognizes the local contextual factors and other 
contributing factors that may undermine or enhance the effectiveness of the WBI training. 

4.3 Conceptual Model 

Our conceptual model for assessing impact recognizes the multi-level structure and multiple 
results of WBI programs. Designed as an intervention in the public policy domain, the results of 
WBI training can be measured over an extended period of time and on different levels. The 
structure and components of the model are illustrated in Annex 1. The model includes: 

• INPUTS include resources used for development and implementation of training 
program. Such resources in addition to the WBI funds may include local in-country 
resources, trust funds, other Bank funds, and funds from other international donors. 

• PROGRAM is a core WBI intervention and a set of logically linked training activities 
that are designed to address a specific areas of public policy. The goals/objectives of the 
programs should adequately address the present conditions in the countries and provide 
the relevant solutions via appropriate combination of the contents, delivery modes, and 
target audiences. 

• IMPLEMENTATION is the actual delivery of a program and is another essential part of 
the intervention. Among the main components are the linkages of WBI programs to other 
intended interventions in the country provided by the Bank, through partnerships with 
international donors and the governments, as well as the technological aspects of 
program delivery, such as distance learning, e-leaming etc. 

• OUTPUTS are the learning gains by the participants resulting from the training received 
and the enhancement of their theoretical and operational knowledge needed for public 
policy design and implementation in various sectors. They may also include increased 
motivation levels and improved attitudes to make changes in improving the economic, 
social and political conditions of their countries (Levels 1 and 2). 

• OUTCOMES are the effects of the learning gains that are manifested through behavioral 
change, performance enhancement and contribute to capacity building, institutional 
change, and sustainability of change. (Level 3). 

• IMPACTS are the result of improved public policy decisions stemming from the 
outcomes of WBI training and positively affected the country’s development indicators. 
(Levels 3 and 4). 

• EXOGENOUS FACTORS (INDIRECT RISKS) are the impediments to the process of 
successful knowledge transmission ( learning process), its utilization {performance 
enhancement) and institutionalization ( improvement of policy) that come from the 
objective conditions in the sector or country. Unfavorable socio-economic conditions and 
political factors in the countries may prevent the utilization of enhanced capacity and 
selection of the best decision for public policy. 
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• EXOGENOUS FACTORS (DIRECT RISKS) emphasize another aspect of 
impediments to the utilization of learning gains: economic, cultural and political 
constraints within existing institutions of public policy that prevent decision making 
within the institutions to benefit from the learning gains and produce more effective 
policy decisions. 

5. Selection of WBI Programs for In-Depth Impact Evaluations in FY02 

WBIES, in consultation with the EILT and WBI managers, selected five of the fifteen WBI 
programs for in-depth impact evaluation in FY02. These programs have been under 
implementation for at least two years and satisfy the following six additional criteria: 

1. Corporate priority (was the program aligned with World Bank Global and Corporate 
Advocacy Priorities?) 

2. Supply/volume (was the program active from FY99-FY01; and number of offerings for 
FY01 as per the CRS) 

3. Demand (number of FY01 participants as per the CRS) 

4. Financial significance (FY02 BB and FY01 secured trust funds) 

5. Instructional modality (did the program use DL?) 

6. Archival data availability (number of FY01 offerings with level-1 evaluations, 
availability of updated participant lists, and on-going impact evaluations) 

To determine the five WBI programs for evaluation, we created an evaluation matrix to assess 
which of the fifteen programs (excluding the new or relatively new programs) merited an impact 
evaluation. We identified eleven characteristics based on the six primary criteria above. These 1 1 
criteria and the evaluation matrix are shown in Annex 2. 

We first ranked programs (1 to 15, with 1 being highest) based on five numerical criteria: the 
number of offerings, the number of participants, FY02 WBI budget, FY01 trust fund budget and 
number of FY01 offerings with level 1 evaluations. To incorporate the non-numerical criteria, 
we rated each program (0 to 1) on corporate priority, instructional modality, availability of 
participant lists and ongoing impact evaluations. We then selected the five WBI programs with 
the highest scores for an in-depth impact evaluation in FY02. 6 These WBI programs are: 

1. Sustainable Development: Natural Resources Management 

2. Good Governance 

3. Decentralization, Fiscal Finance and Urban Management 

4. Education 

5. Health and Population 

For the five programs chosen for impacts, the number of activities within each program from 
FY00-01 was calculated and is displayed in Table 1 below. 




6 Although the Attacking Poverty Program was chosen in our criteria screen for selecting WBI programs for impact, 
it was decided in a WBI review meeting for this framework that the program would be deferred to another year since 
it is too new. 
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Table 1. Total Population of Listed WBI Activities for Impact Evaluation Sampling, By 
Program FY00-01 7 



Program 


Activities FY 00-01 




Non-DL 


DL 


Total 


Health and Population 


43 


25 


<68 


Decentralization 


50 


20 


70 


Education 


4 


0 


4 


Natural Resources 


25 


0 


25 


Governance 


59 


10 


<69 




181 


55 


23<6 



A random sample of approximately 10% of activities will be chosen for impact evaluations from 
the total population of 236 activities. These activities will be a representative sample of the 
activities within the five programs chosen for impacts. 

Annex 3 provides additional detailed information on the program activities for FY01, including 
activities in each of the four quarters. This table also provides more comprehensive summaries of 
the activities within each program for FY01 including the task manger, duration, date and 
number of alumni and non-alumni. Additionally, a pie chart organizes three primary criteria 
(number of participants, number of participant training days and total number of activities) to 
display numerically which programs had the highest numbers for each of the criteria. 

The programs which were not chosen for impacts for FY02 may be selected for evaluation in 
FY03. 



6. Evaluation Strategy and Design 

6.1 Strategy: Program=> Activity 

For each of the five programs, we propose to evaluate a selection of program activities which are 
of a longer duration (such as three days, one week or one month) than the one-time or short 
offerings from which impacts are likely to be negligible or non-existent. Also, we will select 
program activities for which a sufficient amount of time has already elapsed for the desired 
program effects to have manifested themselves. The activities selected for review will be diverse 
in the types of offerings (e.g., core courses, seminars, workshops, study tours, forums) and 
delivery methods (distance learning and face-to-face). Additionally, the country participants will 
represent as diverse a group as possible (e.g., policy makers, civil society and governments). For 
an activity to be selected, a complete and up-to-date participant list must be available. 

In evaluating the impacts of the five WBI programs, we will select a stratified random sample of 
approximately 10% of the activities under each program. We note that there are two basic kinds 
of training activities under each program: those where the majority of key participants come 
from a single country and those where key participants are from several countries (e.g. DL 
courses, conferences). Within this broad stratification, there are training activities with fixed 

7 One-day activities and selected activities with Level 3 evaluations are excluded. 
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durations and others with follow-up activities over an extended period. In selecting program 
activities for evaluation, we will be guided by the need to have a representative number of 
activities that take these distinctions into account. 

In choosing which activities are to be evaluated within the programs, the Evaluation Unit will 
stratify the activities according to five main criteria: 

1. Participant lists. All programs and activities chosen must have an updated and 
complete participant list. 

2. Duration. We will exclude all one day activities, unless they are part of a 
continuous program with similar participants in each activity. 

3. On-going impact studies. WBDES will exclude those activities with on-going or 
completed level 3 evaluations unless there is interest within the WBI program for 
including these activities in the selection process. 

4. Baseline evaluation data. Only those activities with credible level 1 evaluations 
will be selected. 

5. Modality. Where necessary, WBDES will stratify its random selection of activities 
for evaluation in order to be sure that certain training modalities (e.g., DL) are 
represented in the selection. 



6.2 Two-Pronged Approach 

In conducting impact evaluations for the five WBI programs, we propose a two-pronged 
approach: (a) sending electronic tracer evaluation surveys (fax or mailed copies for those without 
electronic resources) and (b) following up with participants through semi-structured face-to-face 
(or phone) interviews and with focus groups in the countries. 

6.3 Design Development 

We will use a program logic model to help guide the development of an effective research design 
strategy. This log frame system will be useful for constructing key elements of the evaluation 
design. It will include: 

• Key evaluation questions and sub-questions; 

• Indicators and measures of key concepts related to the questions; 

• The basic strategy to be employed to answer each question; 

• Identification of data needed and the sources of these data; 

• Methods for collecting the needed data; 

• A preliminary analysis plan. 

6.4 Instrument Development 

We propose the use of pre-tested tracer electronic and fax questionnaires to collect preliminary 
data on WBI program participants. Pre-testing will consist of “desk” testing, where the 
instruments are tested among staff, and field testing, where the instrument is applied in the 
intended context. Results from these tests will help us revise and refine both instrument contents 
and the processes. 
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6.5 Data Sources 

We will use a variety of data sources to answer the key research questions, including from: 

a) Individuals: Program participants (graduates) are the primary sources of data for an impact 
evaluation. Current and complete participant lists, comprising participants’ addresses (including 
e-mail address) and telephone numbers by which they may be contacted, are the means for a 
locating past participants. Random samples of participants will be chosen from these lists. 
Additionally, program managers, regional coordinators in the World Bank and resources persons 
(e.g., facilitators), partners. Country Officers/Economists, and Operational TMs in the area will 
be sought to provide data. 

b) Archival Data: Internal : Program briefs and AIRs, completion reports and BTOs, evaluation 
reports, budget documents, WB country documents, CDF and WB Strategic Framework, CASs, 
and country case studies (internal) are the key archival data sources to be consulted. These will 
be useful for assessing whether the program has met not only its own goals, but also those of the 
WB and the country itself as explicitly stated in these documents. 

External : Literature and relevant research materials on the program, newspaper or magazine 
articles, ministerial directives, country policy reports and other relevant documents (may include 
research through the Web and WB library) are potential sources to check for the program’s 
progress within the country and also to validate the data in the final report (triangulation). 

c) Electronic: Web sites specifically created for the program or those created by program 
managers to maintain contact with participants, for follow-up and for continued networking (may 
include listservs and/or chat rooms) are additional information sources. 

d) Course Materials/ Action Plans 

6.6 Funding 

We will seek donor funding, including from trust funds, to finance this proposed impact 
evaluation work. The BB of the Evaluation Unit will furnish the remaining funding 
requirements. 



7. Data Collection and Analysis 

7.1 Data Collection 



We propose to use mixed methods (qualitative and quantitative methodologies) to assess 
program impacts. These include semi -structured face-to-face and phone interviews and focus 
groups involving program participants and managers, country directors, country office managers, 
regional coordinators, operational TMs, country officials and stakeholders (including partners). 
Random samples of participants will be chosen for the surveys, interviews and focus groups 
from the population of program participants. Some interviews and focus groups will be 
conducted via distance learning technologies through the use of WBI’s Global Distance 
Learning Network in order to minimize costs and travel. 
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Where possible, we will employ trained field consultants in the program countries. These 
consultants will be monitored closely to ensure adequate and high quality responses. Where 
appropriate, we will also use of electronic surveys (questionnaires) to gather basic evaluative 
data. Faxed questionnaires are also possible. We propose to field test all surveys and translate 
them as needed. We will send follow-up letters, e-mails and faxes at intervals of two to three 
weeks after the initial questionnaires are sent to remind participants to complete them. 

There are several purposes to the participant surveys: to remind them of the WBI activity they 
attended; to provide preliminary data on their current employment, addresses, skills they may 
have gained from participation in the activity and the use of their new knowledge in their jobs; 
and to request their consent to be interviewed or to attend a focus group session. Reminding the 
participants of the activity will make it easier to determine the extent of the linkage between the 
knowledge gained through the WBI activity and how they may have used this in their work. 

Where appropriate, we will employ and train country-based consultants and/or staff in WB’s 
country offices in arranging and managing the focus groups and interviews. With the assistance 
of two or three analysts from WBIES, the lead evaluator of the program and additional 
evaluation officers will facilitate the focus groups and conduct the face-to-face interviews. The 
lead evaluator may also task the analysts to conduct interviews themselves, after receiving proper 
instruction and training. Additional interviews and focus groups will be conducted by country- 
based consultants. For face-to-face, telephone and videoconferencing interviews, the lead 
evaluator, consultants and analysts will be responsible for taking notes themselves. For those 
focus groups facilitated by the lead evaluator, an analyst from WBIES or a country consultant 
will be on hand to take notes. 

Country-based consultants will also interview managers and other co-workers of the participants 
to verify the collected data. They will work with WBIES staff in monitoring and collecting 
information on the country, including on government and policy changes from print and local 
media and other sources. These sources will provide additional data reliability and validity 
checks and, thereby, more convincing evidence of impacts. 

WBIES Staff at WB headquarters will also conduct interviews and focus groups with staff 
involved with the WBI program. These will include program managers, regional coordinators, 
country directors and other staff. 

7.2 Data Analysis 

Data analysis will include: 

1) A synthesis of existing evaluations of the program and program activities. 

2) Basic analysis of cost/budget data for each activity. 

3) Analysis/Tracking of participant progression through Kirkpatrick’s four levels of training 
(reaction => learning => behavior => results). 

4) Content analysis of all qualitative data from questionnaires (open-ended questions), 
interviews and focus groups. Emergent themes which detail the main findings will be 
explored through the content analysis. Basic analysis of quantitative data from surveys 
(closed-ended questions) which will include means, standard deviations and percentages. 

O 10 
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5) Analysis of printed program materials (archival data). This includes an examination and 
analysis of: program objectives, invited participants, methods of delivery, needs analysis 
and changes/trends (increases or decreases) in program budget/allocation. 

6) Reliability and validity checks to verify the collected data. Specifically, the use of 
triangulation methods such as printed materials and reports, documented legislation/legal 
reforms, newspaper/magazine articles, case studies/reports written by graduates of the 
program and other relevant printed data to support the notion of program impacts. 

7) Peer review of qualitative data analysis to ensure reliability and validity of the results. 

7.3 Challenges and Constraints 

Since WBI activities are usually implemented simultaneously and/or continually with other 
interventions by the Bank, NGOs, other international donors and development agencies, and by 
governments themselves, singling out the specific areas of impact where the change in 
development indicators can be directly attributed to the effects of WBI programs is a challenge. 
Special attention will be paid to identifying and analyzing the regional and sectoral exogenous 
factors in the countries that have direct effects on the results of the implementation of WBI 
programs. 

Due to the complex nature of the effects of WBI programs and the difficulties of distinguishing 
direct and indirect influences (considering the multiple dimensions of impact and exogenous 
variables), the impact evaluation recognizes several challenges: 

• Defining impacts of the training is difficult due to the multiplicity of intervening 
exogenous factors that may serve as impediments to successful learning, utilization of 
knowledge gains and decision making. 

• Multiple impact indicators will vary across WBI programs, countries and regions, 
reflecting the variations in local conditions and program specific goals and objectives. 

8. Reporting Findings 

The final report will detail key findings of the evaluation. The evaluations will draw upon the 
analyzed data to determine the impact results which the program has had on individuals and 
organizations within participating countries. It will also address the issue of establishing a set of 
benchmark indicators which can be used to monitor this impact over time. 

For each of the five WBI programs, we will report analytical findings for two to three major 
countries where the program’s activities have been delivered. We will also report case studies 
for WBI training activities that cut across counties. All reports will be peer reviewed among staff 
in WBIES. 



9. Dissemination 

We will disseminate all evaluation reports on specific WBI programs to audiences including the 
WBIILT, stakeholders and managers of the program, interested World Bank staff, donors, 
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partners and participants. For evaluation results for which there might be a larger outside 
interest, we will post versions of our results in an external website. 



10. Timetable of Evaluation Activities 

The following table shows a proposed timetable for the impact evaluations. 



Evaluation Activity 


Timing 


Selection of five programs for impact evaluation 


October 2001 


Approval of WBI impact evaluation framework and strategy 


November 2001 


Development and customization of impact strategy for five 
programs 


November - December 2001 


Begin impact evaluation of five programs 


November - December 2001 


- Development of evaluation instruments 


November 2001 - December 2001 


- Data collection 


December 2001 - April 2002 


• Data analysis 


April 2002 - June 2002 


- Completion of draft evaluation reports 


July2002 


- Peer review of evaluation reports 


July 2002 


Completion and dissemination of final evaluation reports 


August 2002 
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Annex 1. Impact Evaluation Framework for WBI Programs 
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EXOGENOUS FACTORS (INDIRECT RISKS) 

- sector/subsector specific factors; 

- regional/national specific factors; 

- global socio-economic and political factors. 



Annex 2. Selection Matrix for WBI Programs 
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